The present invention relates to monitoring the transmission of network data, for example, on the Internet, and, particularly to, a monitoring system that provides improved detection of network anomalies.
Computer networks, such as the Internet, transmit data among computers over a variety of different communication media (e.g., electrical cable, fiber optic cable) joined together by different network switches or routers. Common data transmission protocols, such as TCP/IP, break the data into discrete packets individually routed and assembled at the destination. The data may be from any source that may be converted to a digital form, including text, video and audio material.
With the world's increased reliance on the Internet as a communications link, the monitoring of computer networks, to ensure their proper operation and to respond rapidly to network problems, has become increasingly important. Of particular concern, is the accurate and prompt detection of network “anomalies”, that is, unusual network activity that may signal a problem. Network anomalies may reflect malicious activity such denial of service attacks, where a flood of data packets is directed against a given network node to block its normal function or a broad scale interrogation of a network by a system looking for weaknesses in the network that could be exploited. Network anomalies may also reflect innocent activities that should nevertheless be monitored, including “flash crowd” events occurring because of unexpected and episodic demand for particular data, for example, an unexpectedly popular sporting event sourced from one server to many subscribers, or “node failures” including generally network hardware, network media, or network software causing a significant shift in network traffic and network capacity.
Traffic on particular portions of a network may be monitored by network administrators using a variety of tools allowing automatic and manual monitoring of data collected, for example, from Simple Network Management Protocol (SNMP) queries and “IP flow monitors”. SNMP queries obtain data from network nodes, such as routers, and consist mostly of counts of activity, such as the number of packages transmitted over the node. IP flow monitors provide higher level information about network traffic including the source and/or destination of the data packets, for example, to identify the relationships of packets into logical messages or sessions.
Automating the process of detecting network anomalies is important because of the large amount of network data and the impracticality of constant human monitoring of network events. Nevertheless, this automation process is difficult, particularly given the high variability of normal network traffic. Simple thresholding techniques, when adjusted to limit “false positive” detections, may be unable to detect important anomalies that make minor changes in fundamental network statistics. The use of more complex models, for example, neural nets that model normal network behavior, run the risk of bias toward “known” anomalies at the expense of important unknown or unexpected anomalies. Highly sophisticated automated detection techniques that require large amounts of data storage or computer power, may be impractical for routine network analysis.